Skip to content

sync game data automatically - #2

Merged
lyn1dev merged 2 commits into
mainfrom
auto-sync-data
Nov 7, 2026
Merged

lyn1dev merged 2 commits into
mainfrom
auto-sync-data

Conversation

@lyn1dev

@lyn1dev lyn1dev commented Nov 7, 2026

Copy link
Copy Markdown
Owner

what this does

the site's recipes and fusions were a one-off snapshot from august, and the scripts that built them weren't in the repo, so they never updated. this adds a github action that rebuilds them every 6 hours and publishes the result to main, which deploys on vercel.

data source
fusions, shards SkyShards-Parser (newest available, includes its nov 3 fix)
shard icons SkyShards
craft recipes, enchants, essence shops, slayer xp NotEnoughUpdates-REPO
craft rarity, collection unlocks hypixel api

safety checks

  • broken data (bad format, missing values, html in names) stops the sync and opens an issue. the site keeps the last good data.
  • suspicious changes (over 10% of fusions gone, any shard removed, over 5% of crafts gone or changed) stay off the live site and go to a data-sync-review branch for you to look at.
  • on the site: the sidebar shows when data last synced. a warning appears if syncing stops for 48 hours, or if the bazaar lists shards with no fusion data.

third-party text now reaches the page without review, and that page's localStorage holds api keys, so the sync rejects html characters and enforces strict formats on anything that goes into an html attribute. esc() also escapes quotes now.

wrong info this fixes

compared on the same frozen prices:

  • fusions: 5 of 154 max profit flips relied on a step the current data no longer lists (tadgang + murderwing → mossybit).
  • wrong profits: sepialot and goldolot take 2 shards, not 5, so sepialot + murderwing → tewtil showed 150,703 profit instead of 376,198.
  • missing shards: chocobun, packrat, folf and flora now appear.
  • craft rarity: every craft said common, and 191 of 204 tradable ones weren't. the rarity filters came up empty.
  • slayer levels: the zombie xp table was used for every slayer.
  • recipe count: the banner said 257,195 by counting A+B and B+A separately. it's 131,748.
  • crafts: tradable crafts go from 204 to about 320, including enchanted diamond and enchanted cookie. the old build skipped recipes with a count, recipes stored in .recipes[], and ids like INK_SACK-3.
  • essence shops: now generated, and only recipes you can actually sell on the bazaar get a filter button.
  • unlock requirements: about 500 more crafts now show one, including from the "requires" text. requirements the profile filter can't check (like Enchanting X) are shown on the card but don't hide the craft.

also: bazaar tax is now a number box, defaulting to the old 1.125%. i could only find one guide for the rates (1.25% base, 1.125% with the free upgrade), so please check that's right. "verified recipe ingredients" now just says "recipe ingredients", since we can't verify them.

testing

  • 20 unit tests for the data build and safety checks. these run in the action.
  • 44 browser checks in headless edge on the real site with the new data: all five modules, new shards and icons, rarity and essence filters, multi-output batches, tax, the warnings, slayer levels. throwaway script, not in this pr.
  • 26 checks on publish-sync.sh against a throwaway git remote with a fake gh: publishing, nothing to publish, main moving mid-sync, review, and the issue fallback. also not in this pr.

not tested on github: schedules only run from the default branch, so the publish and review steps can't run until this is merged. this pr only runs the tests and a dry run on github. the review logic was tested locally, as above.

after merging

  1. go to actions → sync game data → run workflow, and check it succeeds and vercel deploys the bot's commit.
  2. optional: enable settings → actions → general → "allow github actions to create and approve pull requests". without it, suspicious changes open an issue with a link to review instead of a pr.

still not guaranteed

hypixel doesn't publish recipes or fusion rules, so these depend on community projects. fusion updates rely on the parser's maintainer running it, and the site can spot new shards but not changed fusion rules. the parser's nov 3 fix itself says "not 100% accurate due to game bug". anvil limits, order caps and 11 slayer unlocks are still maintained by hand. the readme covers this.

items/, collections.json, profit.txt and shard_fusion_profit.py are no longer used by anything, and items/ still gets deployed. i left them out of this pr to keep it reviewable, but they can be deleted.

🤖 Generated with Claude Code

the static data was a one-off snapshot from august and the scripts that built
it weren't in the repo, so it never updated. this adds a sync that rebuilds it
every 6 hours from NotEnoughUpdates-REPO, SkyShards-Parser and the hypixel api.
broken data stops the sync, suspicious changes wait for a review, and everything
else goes live.

the site now shows when data last synced, and warns if syncing stops or the
bazaar lists shards it doesn't know about.

fixes found along the way:
- every craft was labelled common, which also emptied the rarity filters
- slayer levels used the zombie xp table for every slayer
- essence recipes with an unpriced material would have counted it as free
- esc() didn't escape quotes
- bazaar tax is now a setting, since it depends on the community shop upgrade
first run of scripts/sync-data.mjs.

- fusions: 131,748, replacing a list that counted A+B and B+A separately
- new shards: Chocobun, Packrat, Folf, Flora
- fuse amount: Sepialot and Goldolot take 2, not 5
- crafts: real rarities, unlock requirements, and recipes the old build missed
- essence shops now come from NotEnoughUpdates-REPO
@vercel

vercel Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
shard-flipper Ready Ready Preview Nov 7, 2026 11:15am UTC

@lyn1dev
lyn1dev merged commit 2f35c7f into main Nov 7, 2026
3 checks passed

This branch was successfully deployed

1 active deployment
Preview — b82aaf6e Deployed Nov 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant